Capturing test data associated with error conditions in software item testing

ABSTRACT

In a method of testing a software item, an automated test of a software item is conducted. In response to occurrence of an error condition during the test, test data associated with the error condition is captured. A portion of the captured test data is forwarded to a designated electronic notification location.

CROSS REFERENCE TO RELATED U.S. APPLICATIONS

This application is related to U.S. patent application Serial Number ToBe Determined, by David Gibbens et al., filed on Oct. 12, 2012, entitled“SOFTWARE TEST AUTOMATION,” with attorney docket number B049, andassigned to the assignee of the present application; to the extent notrepeated herein, the contents of this related patent application arehereby incorporated herein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined, by David Gibbens, filed on Oct. 12, 2012, entitled“ALTERING A TEST,” with attorney docket number B050, and assigned to theassignee of the present application; to the extent not repeated herein,the contents of this related patent application are hereby incorporatedherein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined, by Callistus Clark Schissel et al., filed on Oct. 12,2012, entitled “SCHEDULED SOFTWARE ITEM TESTING,” with attorney docketnumber B051, and assigned to the assignee of the present application; tothe extent not repeated herein, the contents of this related patentapplication are hereby incorporated herein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined, by David Gibbens et al., filed on Oct. 12, 2012, entitled“SOFTWARE DEVELOPMENT KIT TESTING,” with attorney docket number B052,and assigned to the assignee of the present application; to the extentnot repeated herein, the contents of this related patent application arehereby incorporated herein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined by Sean McDonald et al., filed on Oct. 12, 2012, entitled“CLOUD-BASED SOFTWARE TESTING,” with attorney docket number B053, andassigned to the assignee of the present application; to the extent notrepeated herein, the contents of this related patent application arehereby incorporated herein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined, by David Gibbens et al., filed on Oct. 12, 2012, entitled“TEST CREATION WITH EXECUTION,” with attorney docket number B054, andassigned to the assignee of the present application; to the extent notrepeated herein, the contents of this related patent application arehereby incorporated herein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined, by Mark Tempel et al., filed on Oct. 12, 2012, entitled“TEST ENVIRONMENT MANAGED WITHIN TESTS,” with attorney docket numberB056, and assigned to the assignee of the present application; to theextent not repeated herein, the contents of this related patentapplication are hereby incorporated herein by reference.

This application is related to U.S. patent application Serial Number ToBe Determined, by Mark Tempel et al., filed on Oct. 12, 2012, entitled“TEST LANGUAGE INTERPRETER,” with attorney docket number B057, andassigned to the assignee of the present application; to the extent notrepeated herein, the contents of this related patent application arehereby incorporated herein by reference.

BACKGROUND

Testing often consumes a considerable amount of time when developingsoftware. Typically, a programming background is required to createautomated tests for software applications, application programminginterfaces, software development kits, web services and websites. Mosttest applications require users to learn a specific language to writetests. In addition, when a new instance (build, version, etc.) of aproduct is developed, new or modified tests may be required to ensurethat the new instance of the product performs correctly.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and form a part ofthis specification, illustrate various embodiments and, together withthe Description of Embodiments, serve to explain principles discussedbelow. The drawings referred to in this brief description of thedrawings should not be understood as being drawn to scale unlessspecifically noted.

FIG. 1 is a block diagram of an example testing framework, in accordancewith various embodiments.

FIGS. 2A and 2B are block diagrams of some example computingenvironments with which or upon which embodiments may be implemented.

FIG. 3 is a block diagram of an example library and its contents, inaccordance with an embodiment.

FIGS. 4A-4J are examples of a graphical user interfaces, in accordancewith an embodiment.

FIGS. 5A and 5B illustrate a flow diagram of a method implemented by asystem for creating a custom action, in accordance with variousembodiments.

FIG. 6 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 7 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 8 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIGS. 9A and 9B are block diagrams of some example computingenvironments with which or upon which embodiments may be implemented.

FIG. 10 illustrates an example of a testing task scheduler graphicaluser interface, in accordance with various embodiments.

FIG. 11 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 12 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 13 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 14 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 15 illustrates a flow diagram of a method of software item testing,in accordance with various embodiments.

FIG. 16 is a block diagram of an example testing framework 100B,according to various embodiments.

FIG. 17 is a flow diagram of a method of software development kittesting, in accordance with various embodiments.

FIG. 18 shows an example test framework comprising a test environmentmanagement layer.

FIGS. 19A and 19B are block diagrams of some example computingenvironments with which or upon which embodiments may be implemented.

FIG. 20 is a flow diagram of a method of managing a test environment, inaccordance with various embodiments.

FIG. 21 is a flow diagram of a method of executing a test while creatingthe test, in accordance with various embodiments.

FIG. 22 illustrates an example language evaluator, in accordance withvarious embodiments.

FIG. 23 is a flow diagram of a method of binding a software item to aplain English name, in accordance with various embodiments.

DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to various embodiments, examples ofwhich are illustrated in the accompanying drawings. While variousembodiments are discussed herein, it will be understood that they arenot intended to be limiting. On the contrary, the presented embodimentsare intended to cover alternatives, modifications and equivalents, whichmay be included within the spirit and scope the various embodiments asdefined by the appended claims. Furthermore, in this Description ofEmbodiments, numerous specific details are set forth in order to providea thorough understanding. However, embodiments may be practiced withoutone or more of these specific details. In other instances, well knownmethods, procedures, components, and circuits have not been described indetail as not to unnecessarily obscure aspects of the describedembodiments.

Notation and Nomenclature

Unless specifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present Descriptionof Embodiments, discussions utilizing terms such as “accepting,”“accessing,” “annotating,” “binding,” “capturing,” “checking,”“conducting,” “continuing,” “creating,” “determining,” “embedding,”“executing,” “forwarding,” “implementing,” “indicating,” “initiating,”“invoking,” “maintaining,” “pausing,” “presenting,” “receiving,”“referencing,” “resuming,” “retrying,” “reverting,” “searching,”“starting,” “stopping,” “storing,” “terminating,” “testing,”“traversing,” “validating,” “verifying,” or the like, often refer to theactions and processes of an electronic computing device or system, suchas a virtual machine, among others, of a distributed computing system.The electronic computing device/system transmits, receives, stores,manipulates and/or transforms signals represented as physical(electrical) quantities within the circuits, components, logic, and thelike, of the electronic computing device/system into other signalssimilarly represented as physical electrical quantities within theelectronic computing device/system or within or transmitted to otherelectronic computing devices/systems.

Overview of Discussion

Herein various systems, methods and techniques for providing a testframework are described. As discussed herein, the testing framework isflexible such that tests may be performed on various software items atvarious levels (e.g., a user interface (UI), a web-browser interface, anapplication programming interface (API), etc.) and perform variousoperations (e.g., test for conformity, test for bugs, test operationalreliability, requirement testing, performance testing, etc.).

A framework for integrated test (FIT) is a tool used to create automatedtests. Conventionally, a FIT can be thought of as a stack with a staticframework layer, upon which a dynamic (i.e., customizable) fixture layersits, upon which static test layer sits. With a conventional FIT,programmers write tests for the fixture layer and the fixture layergathers data from HTML files to perform actions. In the conventionalFIT, the fixture layer is tied to a specific instance of a software item(version, build within a version, or the like) and often requireschanging whenever a product revision takes place or a new build of anexisting version of a software item is produced. This programming aspectis a deterrent from adopting a new testing framework, such as aconventional FIT, as the time required to learn a custom programminglanguage (and its syntax) in order to program or alter fixture layers ofa conventional FIT can be considerable. Due to the custom programmingrequirements, many testers hire professional programmers to write testsfor the fixture layer of a FIT for a particular software item undertest. This can be costly. The amount of custom programming andassociated cost and time can also be a deterrent to change betweeninstances (builds and versions) of software items, as more changestypically equate to a greater re-write time for the dynamic fixturelayer used to test the changes.

FIG. 1 is a block diagram of an example new testing framework 100, inaccordance with various embodiments. Testing framework 100 differs froma conventional FIT. Testing framework 100 comprises layers that aresubstantially persistent, as well as at least one layer that is dynamic.For example, testing framework 100 includes a framework layer 110 whichis substantially persistent (also referred to as substantially static).Layers that are substantially persistent are compiled prior to thecustom action layer being run. Thus, substantially persistent layers areless changeable than the custom action layer 140 because they havealready been compiled. Testing framework 100 further comprises a basefixture layer 120 which is substantially persistent, a product fixturelayer 130 which is substantially persistent, a custom action layer whichis dynamic (i.e., changeable), and a test layer 150 which issubstantially persistent after being written. It should be noted thatproduct fixture layer 130 is not required to be included, and may not beincluded in embodiments described herein. In some embodiments, customaction layer 140 interfaces between product fixture layer 130 (whenincluded) and test layer 150. In embodiments where fixture layer 130 isnot included, custom action layer 140 interfaces between base fixturelayer 120 and test layer 150. As will be further described herein,custom action layer 140 can built and/or modified via entry ofparameters into fields of a user interface (e.g., 212 (of FIGS. 2A and2B)). The parameters are typically expressed as natural language words,or as strings of such natural language words. This allows a user toeasily create and modify customized actions in a commonly used languagewith easy to understand syntax and eliminates the need to learn aspecial purpose language simply for programming a fixture layer as wouldbe required in a conventional FIT. Typically, customized actions 320(shown in FIG. 4A) of custom action layer 140 are built/modified foractions which are frequently used in tests of test layer 150 (thusautomating and/or accelerating the creation of those test items) and/orfor actions which are expected to change as software changes.Embodiments, as described herein, are not obvious over conventional FITsbecause they require an extensive and burdensome alteration to theconventional manner that a FIT is structured, and such a change is notan easy, quick, intuitive, or low cost undertaking. In embodimentsdescribed herein, the natural language created as part of the test caseeditor (e.g., custom testing agent GUI 410 of FIG. 4A) is utilized tocreate the additional software layers of the custom actions 320 asopposed to extending existing code (e.g., modifying an existing layer).In other words, one product fixture layer 130 may be added to theconventional FIT model for actions that change infrequently, and acustom action layer 140 is provided to add and modify custom actions 320that are frequently referenced. By adding a layer of abstraction (i.e.,custom action layer 140) to fixture layers (120, 130) tests can haveadditional capabilities and the persons creating tests do not have tore-compile code every time a change is made to the tests.

With continued reference to FIG. 1, custom action layer 140 can be usedto create custom actions 320 that can then be called by tests in testlayer 150. This allows test layer 150 to remain static or relativelystatic from version to version and build to build of a software item,with changes being handled through revisions to the custom actions ofcustom action layer 140. This also allows data formats to be basic andconsistent across all test types for a variety of software items anddifferent instances (e.g., version(s) or build(s) within a version) ofthose software items. Thus, testing framework 100, as described herein,allows a user to write a test for test layer 150 and a product fixturelayer 130 once, and then easily change custom action enabled test caseswithin custom action layer 140 when an instance of a software item undertest 213 (of FIGS. 2A and 2B) changes.

Example Computing Environments

FIGS. 2A and 2B show example computing environments with which or uponwhich embodiments of framework 100 may be implemented. As used herein,“a computing environment” comprises one or some combination ofcommunicatively coupled computing systems. FIG. 2A shows an examplecomputing environment 210A where testing may be performed locally. In anembodiment, a target computing environment 220 (e.g., 220-1, 220-2,220-n) may be accessed by computing environment 210A. Moreover, a targetcomputing environment 220 may be used to access computing environment210A. Embodiments of example computing environment 210A are scalable,and target computing environments 220 may be added and removed fromcomputing environment 210A. Target computer environments 220 eachcomprise user interfaces 212 and the software items under test 213.

In an embodiment, computing environment 210A comprises framework 100,user interface generator 211, user interface 212, and software itemunder test 213. In some embodiments, computing environment 210A furtherincludes a virtual machine generator 214. Although not depicted, itshould be appreciated that computing environment 210A includes one ormore processors and storage (e.g., memory, disk storage, etc.).

Framework 100, as discussed herein, is operable to test a software itemunder test 213 and comprises various layers including a custom actionlayer 140. User interface generator 211 may create a user interface 212for a remote or local user. In other words, user interface generator 211pushes a user interface 212 to a user running a test on the localcomputing environment 210A, or a target computing environment 220. Itshould be understood by those skilled in the art that a target computingenvironment 220 may be a real or virtual machine, a server, a personalcomputer, or any other type of computing environment.

A local user of computing environment 210A may utilize user interface212 to run tests on a software item under test 213. Software item undertest 213 may comprise software local to computing environment 210A orsoftware on a target computing environment 220. Computing environment210A may import some or all of the software item under test 213 from atarget computing environment 220. Lastly, virtual machine generator 214,when included, is operable to create a specified virtual machineplatform in computing environment 210A or a target computing environment220, on which a software item may be tested.

FIG. 2B shows a cloud computing environment 210B. In an embodiment,framework 100 is hosted in a cloud computing environment 210B. Cloudcomputing environment 210B comprises many of the same features ofcomputing environment 210A. For example, cloud computing environment210B is also coupled to target computing environments 220 (e.g., via anintranet, the Internet, or other communicative coupling), and cloudcomputing environment 210B comprises a user interface generator 211,software item under test 213, and a virtual machine generator 214.Although not depicted, it should be appreciated that computingenvironment 210B includes one or more processors and storage (e.g.,memory, disk storage, etc.).

Cloud-based computing environment 210B can be utilized in the samemanner as computing environment 210A. In other words, computingenvironment 210B can be accessed remotely from a target computingenvironment 220 to create/modify portions of framework 100 locatedwithin computing environment 2108 or on a target computing environment220 (which may be local or remote from a user). In an embodiment, a usercan remotely perform all testing in a cloud computing environment 210B.It should be appreciated that access and configuration of one or moreresources of cloud-based computing environment 2108 may be provisionedin response to payment of a fee by a user and/or as a fee based service.

Libraries

FIG. 3 depicts a block diagram of an example library 301. Variouslibraries 301 may be created for various products. As shown in FIG. 3, alibrary 301 comprises at least one custom action enabled test case 315.In an embodiment, a user can create, delete, and edit libraries 301. Inone embodiment, a single library 301 may be used for a plurality ofproducts and/or instances (e.g., version(s) or build(s) within aversion) of a software item. In some embodiments, each library 301 isunique to a particular product and/or instance of a software item.Library 301 may be stored within tangible computer readable storage of acomputing environment 210 (e.g., 210A, 2108) or in a storage thatcommunicatively coupled with a computing environment 210.

In an embodiment, a library 301 may contain one or a plurality (e.g.,two, hundreds, thousands, or more) of custom action enabled test cases315 for a particular product. Each custom action enabled test case 315may comprise a plurality of custom actions 320. Libraries 301, customaction enabled test cases 315, and custom actions 320 may be created,edited, stored, and utilized independently of each other. For example,one custom action enabled test case 315 may be a part of multiplelibraries 301. Similarly, one custom action 320 may be used in aplurality of custom action enabled test cases 315.

Custom Action Layer

With references to FIGS. 1 and 4A-4F, the test framework 100 describedherein comprises a custom action layer 140 operable to run custom actionenabled test cases 315 and custom actions 320. In an embodiment, acustom action layer 140 is positioned between a test layer 150 and atleast one fixture layer (e.g., product fixture layer 130, base fixturelayer 120). In some embodiments, the test layer 150 and product fixturelayer 130 surrounding a custom action layer 140 remain static (i.e.,substantially persistent). As an example, a static test layer 150 maysit upon a dynamic custom action layer 140, which sits upon a staticproduct fixture layer 130, which sits upon a static base fixture layer120, which in turn sits upon a static framework layer 110. Customactions 320 written in the custom action layer 140 are defined in thenatural language of the software item under test 213. In other words,after a programmer writes an interface between a custom action layer anda fixture layer (e.g., 120 or 130), a novice that only knows how towrite custom actions 320 may write custom action enabled test cases 315for the software item under test 213 via input of natural languageinformation and other parameters into fields via one or more graphicaluser interfaces 212 provided by user interface generator 211. GUIs 410,430, and 440 of FIGS. 4A-4F are some examples of such user interfaces212 that may be generated by user interface generator 211.

Custom Action Enabled Test Cases

Still referring to FIGS. 4A-4F, in an embodiment, a custom actionenabled test case 315 comprises one or more lines 425 (actions 421,subjects 422, and values 423) which cause the custom action enabled testcase 315 to perform one or more actions 421. Actions 421 are comprisedof both fixture actions, which are substantially persistent, and customactions, which are dynamic. Custom action enabled test cases 315 can berun automatically (i.e., they may be automated) when invoked by a testfrom test layer 150.

With reference to FIG. 4A, in some embodiments, a custom action enabledtest case 315 can be written simply by entering parameters 426 (seee.g., FIG. 4D) into fields on lines 425. Custom testing agent GUI 410illustrates an example which may be utilized to input parameters 426into lines 425 (e.g., 425A and 425B). In various embodiments, customtesting agent GUI 410 can test lines 425 for syntax errors 450 (as shownin FIG. 4E) to see if they will operate correctly during a test. Itshould be noted that in some embodiments custom actions 320 are executedin response to a user input instructing a check of whether the customaction enabled test case 315, custom action 320 or a sub-portion ofeither (e.g., a line) will execute correctly, while in otherembodiments, a user interface may automatically validate such executionof a line of a custom action 320 or a custom action enabled test case315 in response to completion of entries of all required parameters intothe line. As an example, by inclusion of check marks FIG. 4A shows thatlines 425A, 425C and 425D have validated as being executed successfully,while line 425B (which does not have a check mark) has not executed. InFIG. 4A, message window 416 has been presented following the successfulvalidation of the execution of a row, such as row 425D.

In reference to FIG. 4B, in an embodiment, a new test type GUI 430 iscreated by user interface generator 211 and presented as a userinterface 212 via which a user can create a custom action enabled testcase 315. In some embodiments, at least one base test type 435 isprovided to a user. For example, base test types 435 which custom actionenabled test cases 315 are based upon may include, but are not limitedto: user interface tests 431; web browser tests 432; API tests; softwaredevelopment kit (SDK) tests, a product specific test 433 etc. Asexamples, user interface tests 431 can manipulate, and run tests againsta user interface 212 such as the Windows™ desk top, while web browsertests 432 can test websites and web services using various browsers. Asis illustrated in FIG. 4B, in some embodiments, a user is presented withthe option to create a completely new custom action enabled test case315, or to choose to modify a stored custom action enabled test case 315or base test type 435.

With continued reference to FIGS. 4A-4F, product specific tests 433provide users with additional custom actions 320. As discussed above,product specific tests 433 may also be utilized as a base test type 435.Product specific tests 433 may be unique to a product and may be basedon (i.e., an extension of) another test. In other words, a productspecific test 433 includes all of the fixture actions and custom actions421 in from a base test type 435, as well as additional fixture actionsand custom actions 421 specific to the product test type. As examples, aProtect test is based on a user interface test 431, while testing VMwareGo™ test is based on a web browser test 432. As described herein,products may include, but are not limited to: applications, userinterfaces, APIs, websites, web services, SDKs, custom products, etc. Itshould be understood that a specific product test type 433 may also be abase test type 435 itself (i.e., it is both a specific product test type433 and a base test type 435).

As discussed briefly above, in some embodiments, custom action enabledtest cases 315 may be stored independently of libraries 301 of customaction enabled test cases 315. For example, a single custom actionenabled test case 315 may be used in a plurality (e.g., two, hundreds,thousands, or more) of libraries 301. Thus, by changing a single customaction enabled test case 315, every library containing that customaction enabled test case 315 will have changed. If a custom actionenabled test case 315 is deleted, every library 301 that contained thatcustom action enabled test case 315 will no longer be able to accessthat custom action enabled test case 315. In some embodiments a noticemay be sent to a user indicating that a custom action enabled test case315 has been deleted or cannot be found. Via the custom action layer140, custom action enabled test cases 315 reduce the size of a test(e.g., a test in a test layer 150) from, for example, hundreds orthousands of operations (e.g., operations to be performed at a fixturelayer 120) to a fewer number of custom actions 320. Moreover, thelanguage in which custom actions 320 and custom action enabled testcases 315 are written in is easier to comprehend than a specificprogramming language (e.g., C, C#, Java) used for writing typical testsin a test layer 150. This allows a non-programmer to create or modifymany tests quickly.

Custom Actions

Referring to FIGS. 4A and 4C-4F, collections of actions 421 may bechosen for various products and used in various custom action enabledtest cases 315. In one embodiment, a line 425 (also referred to as astep or row) of a custom action enabled test case 315 comprises threeparameters 426: (1) an action (e.g., a name, a reference to an action421, etc.), (2) a subject 422, and (3) a value 423. A subject 422 refersto an object or software item under test 213, and a value 423 indicateswhat to change the subject 422 to and/or check the subject 422 against.In an embodiment, subjects 422 and values 423 are comma delimited listsof key value pairs. Consider an embodiment where a user enters a customaction 320 as the action. The custom action 320 performs at least oneoperation (or step, command, etc.) on a subject 422. A feature of customactions is that parameters are expressed in natural language type words.Because of this, custom actions 320 are easily maintainable as they donot require code to be written or compiled before being run.

Custom actions 320 may be stored and used independently of custom actionenabled test cases 315 and/or libraries. For example, a user may createand/or edit a custom action 320. In various embodiments, a custom action320 may be modified (e.g., given a description, modifying a name, etc.)via a GUI such as custom action editor 440. While the custom testingagent GUI 410 allows a user to modify custom action enabled test cases315, the custom action editor 440 allows a user to modify custom actions320. A user may store a custom action 320 independently of a customaction enabled test case 315. In one embodiment, when a user is creatinga custom action enabled test case 315, they may simply insert (orincorporate) a stored custom action 320. This allows a user to editevery custom action enabled test case 315 that incorporates a particularcustom action 320 by editing only the particular custom action 320. Insome embodiments, a user will be notified if a custom action 320 hasbeen deleted, edited, or cannot be found.

Custom Editors

Referring again to FIGS. 2A and 2B and also to FIGS. 4A-4F, in anembodiment, a user interface 212 is created by user interface generator211. For example, a user interface 212 may present a custom testingagent graphical user interface (GUI) 410 that allows a user to enter aseries of custom test actions 320 and/or create a custom action enabledtest case 315. GUI 410 of FIG. 4A is one example of such a GUI. Customtesting agent GUI 410 may allow a user to save/store custom actionenabled test cases 315 in a library 301 (FIG. 3), open libraries 301 ofcustom action enabled test cases 315, and run custom action enabled testcases 315.

As illustrated in FIG. 4D and FIG. 4E, custom testing agent GUI 410 alsoallows a user to check a line 425 of a custom action enabled test case315 for syntax errors 450, valid parameters 426, etc. In someembodiments, custom testing agent GUI 410 allows users to run lines 425of a custom action enabled test case 315. In some embodiments, customtesting agent GUI 410, may be executed from a command line or as part ofan automated build process. As illustrated in FIG. 4F, in someembodiments, auto-completion 460 options are employed and presented suchthat when a user types a portion of a parameter 426, the custom testingagent GUI 410 predicts and presents one or a list of parameters 426 theuser is beginning to enter without the user completely typing in aparameter 426.

Altering a Test while a Test is in an Active State

FIGS. 4G-4I illustrate various portions of a test log 477 generated in atest runner 470. As depicted in by the portion of test log 477 shown inFIGS. 4G and 4H, in some cases, a test case 315 will stop when an error476 is found. When this occurs, a user interface 471 is presented withone or more options 479. In some embodiments, user interface 471 maypresent an option 472 to undertake a “fix action” to alter the testwhile it is maintained in the active state. In some embodiments, userinterface 471 may present an option 473 to retry the test from thebeginning (or from some other point between the beginning and theoccurrence of the error condition 476). In some embodiments, userinterface 471 may present an option 474 to resume the test from thepoint of the error condition, without altering the underlying test, thusaccepting the error as a legitimate failure. In some embodiments, userinterface 471 may present an option 475 to terminate the test. It isappreciated that options 472, 473, 474, and 475 may be presented aloneor in combination with one another and/or in combination with otheroptions.

A test case 315 can take hours or days to run. In various embodimentsdescribed herein, when a test case 315 is running and finds an error476, the test case 315 “pauses” in an active state (i.e., a state wheredata created/edited by a test remains available), and allows a user toedit the paused test case 315 by selecting option 472, and then continuerunning the test without restarting test case 315. In some embodiments,a test case 315 is maintained in an active state within a cloudcomputing environment 210B in which the test case 315 is running. Insome embodiments, a state of a virtual machine upon which a test isrunning is maintained while a test is paused. As briefly discussed inthe Custom Action section, it should be understood that in someembodiments, both test cases 315 and custom actions 320 may be modified.If a custom action 320 is modified, all subsequent calls to that customaction 320 will reflect the change. It should further be appreciatedthat a test case 315 may include a custom action and thus be a “customaction enabled test case” or may include no custom actions. As such,error conditions that a user is permitted to alter while a test ismaintained in an active state include errors in custom actions within atest case 315 and errors in portions of a test case 315 which do notinvolve a custom action.

As shown in FIG. 4G, in various embodiments, when an error 476 is foundwhile a test case 315 is running, the test case 315 pauses. In otherwords, when an error is found, a test case 315 stops executing actions,but remains active. In various embodiments, while a test case 315 ispaused, registers, variables, and parameters maintain the state theywere in when the test paused. In some embodiments, all prior testinginformation 478 up to the occurrence of the error condition 476 ismaintained. For example, this information may be maintained and accessedvia a GUI or portion thereof such as test suite 485, which lists alltests which have been executed or are scheduled to be executed.

Error conditions that may cause a test case 315 to pause include, butare not limited to: not being able to find an item of software (e.g.,software item under test 213) at a prescribed location, not being ableto find a network connection, attempting to use an invalid value, notbeing able to access a machine or data, action syntax errors, etc. Thus,it should be appreciated that error 476, illustrated in FIG. 4G, ispresented by way of example and not of limitation and is only one ofseveral types of error conditions that may cause execution of test case315 to pause.

In various embodiments, test case 315 can be run through a GUI such as atest runner 470, or from within a custom testing agent GUI 410. AlthoughFIG. 4G shows a test runner 470 running a test, it should be understoodthat a test case 315 can be run from elsewhere including, but notlimited to: GUIs, a command line, as part of a build process, etc. Invarious embodiments, screen shots 490 (of FIG. 4J) can be capturedduring the running of a test case 315 (e.g., by a test runner 470). Insome embodiments, test runner 470 is operable to create and provide alog 477 of real-time action 421 execution results 476. It should beunderstood that screen shots 490 may be accessed without using ahyperlink.

As is illustrated in FIG. 4G, in various embodiments, when an error 476occurs, a failure prompt 471 may be provided. Example failure prompt 471prompts a user to select a user selectable option 479 of a plurality ofuser selectable options 479, which include, but are not limited to: (1)fix the current action or active test 472; (2) retry the current action473; (3) resume the active test 474; or (4) terminate the active test475. Example failure prompt 471 may also show a detailed description ofthe error 476 that occurred. It should be understood that actionsprovided by failure prompt 471 (e.g., 472, 473, etc.) may be added,edited, created, or removed in various embodiments. Moreover, in anembodiment, a user may have the option 482 to set whether test runnerprompts a user upon the occurrence of an error.

In an embodiment, a user may alter a test by selecting fix an action 472from failure prompt 471. Such selection allows editing while theexecuting test case 315 is paused. In various embodiments, a user mayedit parameters 426, values within variables, software items, etc.,while a test case 315 is paused. Editing may be performed using customtesting agent GUI 410, or another test editor. Custom testing agent GUI410 may, in some embodiments, show a test case 315, in which actions 421were performed successfully and which caused an error. For example, anaction execution error may be indicated by an object such as a red “X”,and a successfully executed action may be indicated by an object such asa green check mark. In some embodiments, the execution of an alterationto a test is validated while a test case 315 is maintained in an activestate. Also, an alteration, entered via a user interface 212 thatfacilitates entry of the alteration and validation of the execution ofthe alteration, may be accepted. For example, a custom testing agent GUI410 is one example of such a user interface that may be utilized toalter a test and employ syntax error checking 450, auto-completion 460,etc.

FIG. 4H shows an example error hyperlink 481 within a log 477. It shouldbe noted that various types of errors 476 exist other than errors causedby custom actions 320. In an embodiment, after clicking the errorhyperlink 481 within the log 477, a screen shot 490 (as illustrated inFIG. 4J) of an error 476 may appear.

In FIG. 4J, an example screen shot 490 is shown in which a GUI labeled“New Machine Group” has appeared. In this example, a custom action 320was attempting to access a GUI labeled “Machine Group.” Thus, in thisexample, an error 476 occurred when the custom action 320 could not findthe GUI “Machine Group,” and a screen shot 490 was captured. In someembodiments, a plurality of screen shots 490 may be captured and stored.For example, a user may wish to view screen shots 490 captured prior toerror 476.

In some embodiments, a user may check for syntax errors 450 whileediting a parameter 426 (e.g., while changing a subject 422 or customaction 421). Because the test is still active (and paused), in variousembodiments, after a user performs an edit, a user is prompted (withfailure prompt 471 or otherwise) to retry, resume, or terminate a testcase 315. In one example, if an error 476 is fixed and the test case 315completes successfully, log 477 may record the number of actions witherrors that occurred and the number of actions that were fixed.

For example, FIG. 4I shows a result summary 484 that comprisesinformation including, but not limited to: the type of machine that thetest was run on, test duration, the start and stop times of the test,the number of test suites (e.g., test cases 315) run, the number of testsuites that were successful, failed, and/or did not run, the number ofactions performed (e.g., did not fail and did not contained anyvalidation), the number of actions that passed (e.g., containedvalidation), the number of actions that were fixed, the number ofactions that failed, etc.

With reference again to FIG. 4G, in an embodiment, a user may retry anaction 473. For example, if an action 320 attempts to perform anoperation on a machine that is not yet available, the user may retrythat action when the machine becomes available, and the test case 315may continue to run.

In one embodiment, a user may resume a test 474. For example, after atest case 315 pauses due to an error 476, a user may try to resume thecustom action enabled test case 315 after waiting for a period of time.In an embodiment, a test may resume at the point where the error 476occurred. In other embodiments, a test may resume at a point prior towhere the error 476 occurred. In one example, after altering a test tofix an error 476, a test may be resumed at the point when the error 476occurred or at a point previous to when the error 476 occurred. In oneembodiment, by resuming a test, a failure is accepted as a validfailure. In an embodiment, the error 476 (or a description thereof) 483is shown by failure prompt 471 and/or log 477.

In one embodiment, a user may terminate a test 475 upon the occurrenceof an error. Terminating a test causes the test to stop running. In someembodiments, the active state is maintained after a test case 315 isterminated.

Example Methods of Use

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIGS. 5A,5B, 6, 7 and 8, flow diagrams 500, 600, 700 and 800 illustrate exampleprocedures used by various embodiments. Flow diagrams 500, 600, 700 and800 include some procedures that, in various embodiments, are carriedout by a processor under the control of computer-readable andcomputer-executable instructions. In this fashion, procedures describedherein and in conjunction with flow diagrams 500, 600, 700, and 800 are,or may be, implemented using a computer, in various embodiments. Thecomputer-readable and computer-executable instructions can reside in anytangible computer readable storage media. Some non-limiting examples oftangible computer readable storage media include random access memory,read only memory, magnetic disks, and optical disks, any or all of whichmay be employed with a computing environment (210A, 210B, 220). Thecomputer-readable and computer-executable instructions, which reside ontangible computer readable storage media, are used to control or operatein conjunction with, for example, one or some combination of processorsof the computing environment (210A, 210B, 220). It is appreciated thatthe processor(s) may be physical or virtual or some combination thereof(it should also be appreciated that a virtual processor is implementedon physical hardware). Although specific procedures are disclosed inflow diagrams 500, 600, 700, and 800, such procedures are examples. Thatis, embodiments are well suited to performing various other proceduresor variations of the procedures recited in flow diagrams 500, 600, 700,and 800. Likewise, in some embodiments, the procedures in flow diagrams500, 600, 700, and 800 may be performed in an order different thanpresented and/or not all of the procedures described in one or more ofthese flow diagrams may be performed. It is further appreciated thatprocedures described in flow diagrams 500, 600, 700, and 800 may beimplemented in hardware, or a combination of hardware with firmwareand/or software. In some embodiments, one or more of the proceduresdescribed in flow diagrams 500, 600, 700, and 800 may be implemented inresponse to payment of a fee and/or implemented within or via acloud-based computing environment in response to payment of a fee and/oras part of a fee based service.

FIG. 5A is a flow diagram 500 of a method of creating a custom action,in accordance with various embodiments. A custom action can be acommand, action, operation, etc. As discussed above, custom actions 320may be incorporated into custom action enabled test cases 315.

At procedure 510 of flow diagram 500, in one embodiment, user interfacegenerator 211 presents a custom testing agent user interface 212 (e.g.,GUI 410 of FIG. 4A) for entry of a custom action 320. As has beendiscussed, in various embodiments, user interface generator 211 may beimplemented as a portion of computing environment 210A or as a portionof cloud computing environment 2108.

In an embodiment, a custom testing agent GUI 410 allows a user to write,test, and/or execute a step of a custom action enabled test case 315. Acustom action enabled test case 315 may be written by entering an action(see e.g., FIGS. 4A-4G) 421, a subject 422 and a value 423 into a line425 of a custom action enabled test case. In some embodiments, actions421, subjects 422 and values 423 may be chosen from a list 424. In otherembodiments, auto-completion options 460 are employed such that when auser types a portion of a parameter 426, the custom testing agent GUI410 predicts one or more parameters 426 the user is beginning to enterwithout the user completely typing in the parameter 426.

In various embodiments, separate fields are provided for the entry ofparameters 426. In an embodiment, a first field for entry of an action421 to be performed is presented. For example, custom testing agent GUI410 comprises a space to enter an action 421. A second field for entryof subject 422 (e.g., a software item under test 213) on which theaction 421 (i.e., operation) is to be performed is presented. Forexample, custom testing agent GUI 410 comprises a space to enter asubject 422. A third field for entry of a value 423, which is to betested against or changed based upon the action 421 entered into thefirst field, is presented. For example, custom testing agent GUI 410comprises a space to enter a value 423.

At procedure 511 of flow diagram 500, in one embodiment, responsive toreceipt of a sub-portion of a line 425 of the custom action 320 via theuser interface 212, the sub-portion is automatically checked for errorsin syntax 450. Additionally or alternatively the sub-portion may bechecked for auto-completion options 460. For example, while entering aparameter 426, a yellow exclamation mark will appear as the user typesand highlight the syntax errors 450, missing parameters 426, invalidparameters 426, and/or unnecessary parameters 426 present during eachcharacter typed to verify whether the line 425 (i.e., custom action 320)will operate correctly when run. While a user is creating a customaction enabled test case 315, when a syntax error is found, parametersare missing, or parameters are incorrectly included, a user may hoverover an error and a description of an error may appear. In oneembodiment, parameters 426 that are invalid are highlighted. Forexample, an invalid parameter may be shown in red text. In anembodiment, a line 425 refers to a line of code, for example, a customaction 320, subject 422, and value 423. In various embodiments,auto-completion options 460 are presented. For example, selectableparameter auto-completion options 460 may be provided to a user when theuser enters a sub-portion of a parameter 426.

At procedure 512 of flow diagram 500, when included, in one embodiment,responsive to a completed entry of the line 425 via the user interface212, the execution of the line 425 of the custom action 320 is validatedfor proper execution without errors. The validation may be implementedautomatically in response to completion of entry of all parameters onthe line or in response to a user request for such validation. Forexample, once all parameters 426 are entered, a custom testing agent GUI410 may automatically check that the line 425 will operate correctlywhen run. In another embodiment, line 425 can be validated and executedindividually after creation.

At procedure 513 of flow diagram 500, when included, in one embodiment,the custom action 320 is stored as a selectable action 421 in a dynamiclayer of a testing infrastructure which is associated with a particularinstance (e.g., version or build within a version) of a software itemunder test 213. For example, users may store custom test actions 320,custom action enabled test cases 315, and libraries to be run by acustom action layer 140 comprised within a framework 100. In anembodiment, custom actions 320, custom action enabled test cases 315,and libraries may be stored on a user owned computer system. A userowned computer system may be a target computing environment 220, or acomputing environment 210A that includes testing framework 100. In someembodiments, any authorized user owned computer system may add, delete,or edit custom actions 320, custom action enabled test cases 315, and/orlibraries of custom action enabled test cases 315. In anotherembodiment, custom actions 320, custom action enabled test cases 315,and libraries can be added, deleted, edited, and stored on a cloud-basedcomputing environment 210B by an authorized user.

At procedure 520 of flow diagram 500, when included, in one embodiment,a second user interface in which the stored custom action is availablefor selection as an executable testing action in a static layer of atesting infrastructure is presented. For example, user interfacegenerator 211 can generate a user interface for entry of a test for testlayer 150 of framework 100, and a stored custom test action 320 may beentered as a line of the test, and then executed at completion of entryand/or whenever the test is run.

At procedure 530 of flow diagram 500, when included, in one embodiment,a user modification of the stored custom action 320 is accepted at atime after storing of the stored custom action 320. In other words, auser may edit a custom action 320, a custom action enabled test case315, or a library 301 after it has been saved.

FIG. 6 is a flow diagram 600 of a method of software item testing, inaccordance with various embodiments.

At procedure 610 of flow diagram 600, in one embodiment, in response tooccurrence of an error condition 476 during an automated test of asoftware item, the test is maintained in an active state. In oneembodiment, a test (e.g., a custom action 320, a custom action enabledtest case 315, a test fixture, etc.) is maintained in an active statewithin a computing environment 210A, 210B in which the test is running.In one embodiment, a state of a virtual machine (e.g., 210A, 220, etc.)upon which a test is running is maintained.

In an embodiment, maintaining the test in an active state refers to whenthe test is paused at the point in the test where the error condition476 occurred. For instance, when an error 476 occurs a test is paused sothat the test may be resumed without restarting the test. In oneembodiment, an alteration to the test is accepted while the test is inthis paused state. For example, a custom testing agent GUI 410 oranother means of data entry may be used to alter a test while the testis paused.

In one embodiment, testing of the software item then continues using thealtered test. In other words, a test resumes (i.e., continues) from thepoint at which it was paused in an active state, such as at the pointwhere an error 476 occurred. In other embodiments, a test may resumefrom a point prior to the error 476 occurring. In one embodiment, allprior testing information up to the occurrence of an error condition 476is maintained. As an example, data changed or used during the test maybe maintained when a test is paused, or after a test is terminated.

At procedure 620 of flow diagram 600, an alteration to a test isaccepted. For example, a custom testing agent GUI 410 or another meansof data entry may be used to receive an alteration to the test while thetest is paused. In various embodiments, an execution of an alteration toa test is validated while the test is maintained in an active state. Forexample, while the test is maintained in an active state, a user mayalter a test (e.g., via custom testing agent GUI 410) and check whethera test will perform correctly (e.g., check for syntax errors 450,conformity, successful execution, etc.). In various embodiments, analteration to a test is accepted via a user interface 212 whichfacilitates entry of the alteration and validation of the execution ofthe alteration. In one embodiment, an alteration comprising a change toa custom action 320 of a custom action layer 140 of a testing framework100 is accepted. In some examples, a custom action 320 may be added,removed, deleted, or altered. In some examples, a test may be fixed,wherein the test does not comprise custom actions 320. In an exampleembodiment shown in FIG. 4G, a failure prompt 471 is provided thatallows a user to perform operations on a test.

At procedure 630 of flow diagram 600, the test is continued using thealtered test. As discussed herein, in one embodiment, a test iscontinued from a point in the test where the error 476 occurred. Inanother embodiment, a test is continued from a point in the test priorto when the error 476 occurred. A user may be able to choose where tocontinue a test from, in some embodiments. For example, a system maystore screen shots 490, and a user may view the stored screen shots 490.The screen shots 490 may assist a user in determining what point in atest to continue the test from.

FIG. 7 is a flow diagram 700 of a method of software item testing, inaccordance with various embodiments.

At procedure 710 of flow diagram 700, in one embodiment, an automatedtest of a software item using a test (e.g., a custom action enabled testcase 315, a custom action 320, etc.) of a testing framework 100 isinitiated.

At procedure 720 of flow diagram 700, in one embodiment, a test within acomputing environment (e.g., 210A, 210B, 220) in which the test is beingconducted is paused. For example, in one embodiment, a test ismaintained in an active state within a cloud computing environment 210Bin which the test is running. In one embodiment, all prior testinginformation up to the occurrence of the error condition 476 ismaintained while a test is paused. As examples, prior testinginformation includes, but is not limited to: data in memory, valueswithin variables, parameters 426, log 477 information, etc.

At procedure 730 of flow diagram 700, in one embodiment, an alterationto a test is accepted while the test is paused. For example, a customtesting agent GUI 410 or another means of data entry may be used toalter a test while the test is paused. In one embodiment, the portion ofthe test which is altered, by accepting the alteration, is not a customaction 320 (for example the test may not even include a custom action320). In another embodiment, the accepted alteration does comprise achange to a custom action 320 of a custom action layer 140 of a testingframework 100. When the custom action 320 is altered, every instance ofthe test which uses that alteration will immediately be altered in thesame manner when the altered custom action is utilized. In this manner,an error is corrected only one time instead of many times. This reducestime in correcting such errors and also eliminates additional testinginterruptions for the same error condition.

At procedure 740 of flow diagram 700, in one embodiment, a test iscontinued using the altered test. In one embodiment, a test is continued(i.e., resumed) from a point in the test prior to where an error 476occurred while, in another embodiment, a test is continued from thepoint in the test where the error 476 occurred. In other embodiments,the test may be continued from before the point in the test where theerror occurred.

FIG. 8 is a flow diagram 800 of a method of software item testing, inaccordance with various embodiments.

At procedure 810 of flow diagram 800, in one embodiment, a test ispaused within a computing environment in which the test is beingconducted in response to an occurrence of an error condition 476 duringan automated test of a software item.

At procedure 820 of flow diagram 800, in one embodiment, a plurality ofuser selectable options 479, with respect to a paused test, arepresented.

At procedure 830 of flow diagram 800, in one embodiment, an action withrespect to a paused test is implemented in response to receipt of aselection of one of a plurality of user selectable options 479. In oneembodiment, the action is terminating the test. As an example, a usermay select an option 475 via a GUI to terminate the test. Aftertermination, the test cannot be continued and instead must be restarted.In one embodiment, the action comprises resuming a test from a point ofan error 476 while accepting the error 476 as a legitimate issue in thesoftware item under test. For example, in some embodiments, onlyparticular types of errors 476 will cause a test to pause, and upondetermining the cause of the error a user may elect to continue the testwithout altering the test.

In one embodiment, the action comprises retrying a test action thatcaused an error condition 476 to occur, in an attempt at reproduction ofan error condition 476. For example, the error 476 may have occurredbecause it was waiting for access to a machine or data that did notbecome available in time. In that example, retrying may allow the testto run without errors after a test can access a machine or data. Inother embodiments, a user may fix a test and retry the test from thebeginning, or from a particular point in the test.

In one embodiment, a user interface 212 in which a portion of a test maybe altered is presented while the test remains paused. For example, auser may be presented with a custom action testing GUI 410, a testeditor, or some other type of user interface 212 with which to fix atest.

Scheduled Software Item Testing

FIGS. 9A and 9B are block diagrams of example computing environmentswith which or upon which embodiments may be implemented.

FIG. 9A shows an example task scheduling computing environment 910Acomprising embodiments of computing environment 210A, and a taskscheduler 915. FIG. 9B similarly shows a task scheduling cloud-basedcomputing environment 910B comprising embodiments of cloud-basedcomputing environment 210B, with a task scheduler 915, and may performsome or all the processes and operations task scheduling computingenvironment 910A can perform. It should appreciated that access to oneor more resources of task scheduling cloud-based computing environment910B may be provided in response to payment of a fee by a user and/or asa fee based service. With a task scheduling computing environment, theoperational status of an environment or an application is measured usingtesting infrastructure 100, as discussed above.

In an embodiment, task scheduling computing environment 910A/taskscheduling cloud computing environment 910B is operable to performcontinual testing. When testing on staged environments, issues existingon the live environment can go unnoticed. For example, a scheduledbackup may not appear as a possible error in a staged testingenvironment. Task scheduling computing environment 910A/task schedulingcloud computing environment 910B is operable to perform tests and verifythe operational status of a production infrastructure prior to, and postdeployment (i.e., tests continue to be performed post deployment, whichmay help in determining the cause of errors in a non-stagedenvironment). Moreover, task scheduling computing environment 910A/taskscheduling cloud computing environment 910B may send the test results toa specified electronic notification location or locations (e.g., one ormore of an electronic mail address/addresses, a social network, a wiki,a flat file, a spread sheet, a network storage location, etc.). Forexample, task scheduling computing environment 910A/task schedulingcloud computing environment 910B is operable to send a screen shot 490,log 477, or some other documentation of an error condition. Byconducting a test with a testing framework 100 at a particular time(e.g., by using a task scheduler 915), potentially intermittent issuesmay be recognized and reported.

In an embodiment, a task scheduler 915 initiates a test, also referredto herein as an operational check, at specified intervals. For instance,a task scheduler may initiate a test every ten minutes or every twohours (e.g., a recurring test). In one embodiment, the test is initiatedimmediately, at a scheduled time in the future, or at a recurrenceinterval (e.g., every 10 minutes, every 2 hours, etc.).

Not only is a test initiated by the task scheduler able to check whethera machine is operational, it also has the flexibility of the testingframework 100 and its layers (e.g., fixtures 120 and 130, custom actionlayer 140, and test layer 150). This way, various test types 435 such asuser interface tests 431, web browser tests 432, and product specifictests 433 may be run. It should be appreciated that the tests written toperform these scheduling and monitoring functions are composed in thesame commonly used language as test cases 315. In some embodiments, taskcomputing environment 910A/task scheduling cloud computing environment910B is operable to verify certificates. For example, if Microsoft™ TaskScheduler is being used as task scheduler 915, task scheduling computingenvironment 910A/task scheduling cloud computing environment 9108 canverify certificates for the Internet Information Services' current tool.

Various embodiments of the testing framework 100 discussed in thesections above may be scheduled. For instance, in response to an errorduring a scheduled test, the test may be paused within a task schedulingcomputing environment 910A/task scheduling cloud computing environment910B. That is, in one embodiment, a test is paused when an error 476occurs. In one embodiment the task scheduler 915, or other portion ofthe computing environment which is managing a test, can sendnotification to a specified electronic notification location orlocations (e.g., one or more of an electronic mail address/addresses, asocial network, a wiki, a flat file, a spread sheet, a network storagelocation, etc.) while the test is paused. For example, this may comprisesending an electronic message (email) to an electronic mail addresswhile the test is maintained in an active, or paused, state.Additionally, in some embodiments, a single computing environment withinwhich a test is to be performed is specified. Similarly, a plurality ofcomputing environments within which a test is to be performed isspecified.

Task scheduling computing environment 910A/task scheduling cloudcomputing environment 910B can be viewed as a monitoring tool and atesting tool. I.e., in addition to executing recurring tests to checkwhether an environment is running, collections of these test results canbe accumulated over time. That is to say, the recorded history of testsexecuted on a recurring basis itself provides a unique measurement offunctionality and performance measurements sampled over time. This timesampling is unique and provides insights into patterns or periods ofinstability in an environment.

FIG. 10 illustrates an example of a testing task scheduler graphicaluser interface 1000, in accordance with various embodiments. GUI 1000 isone example of a GUI 212 produced by task scheduler 915 and/or graphicaluser interface generator 211 for the purpose of soliciting and receivinguser input to task scheduler 915. It is appreciated that GUI 1000 isprovided by way of example and not of limitation and that in otherembodiments, a testing task scheduler GUI may be utilized which has agreater or lesser number of features than are illustrated in GUI 1000.GUI 1000 provides test name field 1010 in which a user can enter orselect the name of a software item test to be scheduled for initiation.A user may type in the name or select from a list of available tests.GUI 1000 provides a computing environment(s) field 1020 in which a usercan enter or select the name of computing environment on which a test ofa software item is to be carried out. It is appreciated that onecomputing environment may be selected or a plurality of computingenvironments may be selected (e.g., via a comma separated list). GUI1000 provides a plurality of selections (1030, 1040, or 1050) withrespect to test time initiation. For example, by selecting the radiobutton next to the word “Now”, a test will be initiated immediately uponsaving the scheduled test; by selecting the radio button next to theword “once” and entering a date and time into associated fields, a testwill be scheduled for initiation at a specified future time; and byselecting a radio button next to the words “recurring test interval”, auser may specify a recurrence interval for initiating an immediate orfuture test. In some embodiments, a user may be provided with anelectronic notification entry field 1060 in which one or more specifiedelectronic notification locations (e.g., one or more of an electronicmail address/addresses, a social network, a wiki, a flat file, a spreadsheet, a network storage location, etc.) may be entered. For example, aplurality of electronic mail addresses and or other electronicnotification locations may be entered and specified as places to whichto send testing results (e.g., test logs, error reports, screen shots,and the like). After appropriate information is entered a user may clickthe save button 1070 to save button to save the scheduled test. In oneembodiment, clicking save button 1070 will cause the scheduled test tobe listed in test suite 485 (see e.g., FIGS. 4G-4I). In one embodiment,a user may select cancel button 1080 to close GUI 1000 withoutscheduling or altering a scheduled test of a software item.

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIGS. 11and 12 and, flow diagrams 1100 and 1200 illustrate example proceduresused by various embodiments. Flow diagrams 1100 and 1200 include someprocedures that, in various embodiments, are carried out by a processorunder the control of computer-readable and computer-executableinstructions. In this fashion, procedures described herein and inconjunction with flow diagrams 1100 and 1200 are, or may be, implementedusing a computer, in various embodiments. The computer-readable andcomputer-executable instructions can reside in any tangible computerreadable storage media. Some non-limiting examples of tangible computerreadable storage media include random access memory, read only memory,magnetic disks, and optical disks, any or all of which may be employedwith a computing environment (910A, 910B, 220). The computer-readableand computer-executable instructions, which reside on tangible computerreadable storage media, are used to control or operate in conjunctionwith, for example, one or some combination of processors of thecomputing environment (910A, 9108, 220). It is appreciated that theprocessor(s) may be physical or virtual or some combination thereof (itshould also be appreciated that a virtual processor is implemented onphysical hardware). Although specific procedures are disclosed in flowdiagrams 1100 and 1200, such procedures are examples. That is,embodiments are well suited to performing various other procedures orvariations of the procedures recited in flow diagrams 1100 and 1200.Likewise, in some embodiments, the procedures in flow diagrams 1100 and1200 may be performed in an order different than presented and/or notall of the procedures described in one or more of these flow diagramsmay be performed. It is further appreciated that procedures described inflow diagrams 1100 and 1200 may be implemented in hardware, or acombination of hardware with firmware and/or software. In someembodiments, one or more of the procedures described in flow diagrams1100 and 1200 may be implemented in response to payment of a fee and/orimplemented within or via a cloud-based computing environment inresponse to payment of a fee and/or as part of a fee based service.

FIG. 11 is a flow diagram 1100 of a method of software item testing, inaccordance with various embodiments.

At procedure 1110 of flow diagram 1100, in one embodiment, a graphicaluser interface is presented for scheduling a test of a software item. Inan embodiment, graphical user interface generator 211 and/or a taskscheduler 915, within a computing environment 910A/910B operate toprovide a user interface 212 (e.g., GUI 1000) comprising testinformation and/or actions. This GUI may be utilized to receive userinput for scheduling initiation of a test of a software item.

At procedure 1120 of flow diagram 1100, in one embodiment, specifiedinformation regarding conduct of a test is received via the presentedgraphical user interface. The specified information includes a test timefor initiating the test. In one embodiment, a task scheduler 915initiates a test according to a received test time. The specified testtime may be an immediate time, a time in the future, or a recurrenceinterval upon which the test time is to recur after the scheduled time.For example, a GUI such as GUI 1000 allows a user to specify whether atest will be performed immediately, sometime in the future, or at arecurring time (e.g., every ten minutes, two hours, daily, weekly,monthly, etc.).

In various embodiments, specified information regarding the conduct of atest which is received via the GUI may include a selection of a singlecomputing environment within which a test is to be performed on asoftware item or a selection of a plurality of computing environmentswithin which a test is to be performed on a software item. The specifiedinformation may include one or more specified electronic notificationlocations (e.g., one or more of an electronic mail address/addresses, asocial network, a wiki, a flat file, a spread sheet, a network storagelocation, etc.) to which testing results for the scheduled test are tobe sent.

At procedure 1130 of flow diagram 1100, in one embodiment, the test isinitiated at the test time that was received via the graphical userinterface. For example, a test is initiated by a task scheduler 915 at atime indicated by the specified time received. In an embodiment, thisensures that the operational status of an environment under test ischecked. It should be appreciated that an environment under test may bea test environment or a live production environment.

At procedure 1140 of flow diagram 1100, in one embodiment the method asdescribed in procedures 1110-1130, further includes sending results ofthe test to one or more specified electronic notification locations(e.g., one or more of an electronic mail address/addresses, a socialnetwork, a wiki, a flat file, a spread sheet, a network storagelocation, etc.) in response to completion of the test. In an embodiment,a testing log 477 (see, e.g., FIGS. 4G-4I) or portion thereof (e.g., aderivation of a log 477 such as a summary), may be sent to the specifiedelectronic notification location(s). In some embodiments, the methoddescribed in procedures 1110-1130 can further send multiplemessages/results to at least one specified electronic notificationlocation.

At procedure 1150 of flow diagram 1100, in one embodiment the method asdescribed in procedures 1110-1130, further includes sending a report ofan error condition (if an error condition occurs during testing) to oneor more specified electronic notification locations (e.g., one or moreof an electronic mail address/addresses, a social network, a wiki, aflat file, a spread sheet, a network storage location, etc.) in responseto an error condition occurring during the test. For example, in anembodiment, a screen shot, such as screen shot 490, of the errorcondition (if an error condition occurs) is sent to the specifiedelectronic notification locations.

At procedure 1160 of flow diagram 1100, in one embodiment, the method asdescribed in procedure 1150, further includes maintaining the test in anactive state without terminating the test in response to the occurrenceof an error condition. For example when an error occurs, a test may bepaused while an email is sent and may stay paused for a pre-specifiedperiod of time to permit a user time to access the test for purposes ofaltering and continuing the test, or debugging the environment in thestate of the error condition. As discussed above, maintaining a test inan active state may comprise storing various register values, priortesting information, data, etc., such that a test may be resumed.

FIG. 12 is a flow diagram 1200 of a method of software item testing, inaccordance with various embodiments.

At procedure 1210 of flow diagram 1200, in one embodiment, an automatedtest of a software item is initiated at a prescheduled time. Asdiscussed above, in an embodiment, a test may be initiated by a taskscheduler 915 such as Microsoft™ Task Scheduler. GUI 1000 represents anon-limiting example of a GUI which may be presented by a user interfacegenerator 211 and utilized to receive user input to task scheduler 915.In some embodiments, in addition to being scheduled for initiation at aparticular time, an automated test may be scheduled to be initiated on arecurring basis after the prescheduled time.

In some embodiments, an automated test may be initiated in accordancewith direction from a task scheduler 915. In an embodiment, for example,task scheduler 915 initiates an automated test of a software item withina preselected computing environment at a preselected time. Thepreselected test time may include, but is not limited to: when aspecific system event occurs, at a user specified time/date, at a userspecified time/date and on a user selected recurring interval, when amachine (or machines) enters an idle state, when a system is booted,when a user logs on, when a Terminal Server session changes state, etc.

At procedure 1220 of flow diagram 1200, in one embodiment, the test ispaused within a computing environment in which the test is beingconducted in response to occurrence of an error condition during thetest. By pausing the test in response to occurrence of an errorcondition during the test, certain actions may be performed before thetest is resumed. For instance, while a test is paused, in response to anerror condition during a test, a report of the error condition may besent to a specified electronic notification location or locations (e.g.,one or more of an electronic mail address/addresses, a social network, awiki, a flat file, a spread sheet, a network storage location, etc.). Inone example, all prior testing information is maintained up to theoccurrence of the error condition. In one embodiment, a test ismaintained in an active state within a computing environment 910A, 910B,220 in which the test is running. Pausing the test may facilitate a useraltering a portion of the test and resuming the test using the alteredversion of the test.

At procedure 1230 of flow diagram 1200, in one embodiment, results ofthe test are sent to a specified electronic notification location orlocations (e.g., one or more of: an electronic mail address/addresses, asocial network, a wiki, a flat file, a spread sheet, a network storagelocation, etc.) in response to completion of the test. For example, ascreen shot 490, a log 477 (or portion thereof), or other informationmay be sent to an electronic mail address after a test completes, or isterminated. In various embodiments, results of the test includesummaries of actions performed or modification to tests made during therunning of a test, and such summaries may be sent to an electronicnotification location as part of a log 477 or independent of a log 477.

Capturing Test Data Associated with Error Conditions in Software ItemTesting

In various embodiments described herein when an error condition, such aserror 476, occurs during a test of a software item, one or more of avariety of actions can automatically be taken in order to capture testdata that is associated with the error condition. The nature ofaction(s) and the nature of the test data to be captured may vary, insome embodiments, based on the particulars of the actual software itemunder test.

As has been previously described with respect to FIGS. 4G-4J, achronological log 477 is maintained of testing actions performed duringa test of a software item 213. In some embodiments, when an error 476occurs during a test of a software item 213, log 477 is annotated (seee.g., annotation 483 in FIGS. 4G and 4H) in a manner which describes theerror which has occurred. It is appreciated that this annotation appearsin the chronological order of where the error occurred during thetesting.

In some embodiments, as illustrated in FIG. 4I, test summary information484 (which may include a variety of statistics about the test) may beincluded in log 477, or in some embodiments, may be generated separatelyof log 477. For example, FIG. 4I shows a result summary 484 that maycomprise information including, but not limited to: the type of machinethat the test was run on, test duration, the start and stop times of thetest, the number of test suites (e.g., test cases 315) run, the numberof test suites that were successful, failed, and/or did not run, thenumber of actions performed (e.g., did not fail and did not contain anyvalidation), the number of actions that passed (e.g., containedvalidation), the number of actions that were fixed, the number ofactions that failed, etc.

In some embodiments, as has been previously described, a screen shot maybe captured of an execution state of software item 213 upon theoccurrence of the error condition while a test (e.g., a test 315) isbeing run on a software item under test 213. It is appreciated that thetest may be paused while the screen shot is captured. Screen shot 490 ofFIG. 4J, illustrates one non-limiting example of such a screen shot.Screen shots such as screen shot 490 are automatically captured when asoftware item under test 213 lends itself to documenting an errorcondition through such a screen shot capture. That is, screen shotcapture is associated with errors that occur during the testing of GUIapplications in a graphical operating system such as Microsoft's Windowsoperating system, or Web applications that operate through a Web browsersuch as Google Chrome, Mozilla Firefox, Microsoft Internet Explorer,Apple Safari, Opera and the like. Such screen shots can facilitate erroranalysis with respect to a test. At the same time, a log from thesoftware item under test may be captured, or at least a snap shot of itmay be captured, at the time of occurrence of the error condition.

Similarly, when a test case 315 is being run against a software itemunder test 213 which does not lend itself to screen shot capture (i.e.,the software item or portion of the software item under test does notinvolve a user interface), a software item log file from the operationof the software item may be captured, or at least a snap shot of thesoftware item log file may be captured, at the time of occurrence of theerror.

In some embodiments, a hyperlink may be embedded in a test file at thepoint of occurrence of or associated with an error in the chronologicallisting of the actions performed and logged during a test of a softwareitem. The hyperlink may be a link to additional captured data regardingthe error condition such as, but not limited to, a screen shot of a GUI,a snap shot of a software item log, or a combination of a screen shotwith log files.

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIGS. 13,14, and 15, flow diagrams 1300, 1400, and 1500 illustrate exampleprocedures used by various embodiments. Flow diagrams 1300, 1400, and1500 include some procedures that, in various embodiments, are carriedout by a processor under the control of computer-readable andcomputer-executable instructions. In this fashion, procedures describedherein and in conjunction with flow diagrams 1300, 1400, and 1500 are,or may be, implemented using a computer, in various embodiments. Thecomputer-readable and computer-executable instructions can reside in anytangible computer readable storage media. Some non-limiting examples oftangible computer readable storage media include random access memory,read only memory, magnetic disks, and optical disks, any or all of whichmay be employed with a computing environment (210A, 210B, 220, 910A,910B). The computer-readable and computer-executable instructions, whichreside on tangible computer readable storage media, are used to controlor operate in conjunction with, for example, one or some combination ofprocessors of the computing environment (210A, 210B, 220, 910A, 910B).It is appreciated that the processor(s) may be physical or virtual orsome combination thereof (it should also be appreciated that a virtualprocessor is implemented on physical hardware). Although specificprocedures are disclosed in flow diagrams 1300, 1400, and 1500, suchprocedures are examples. That is, embodiments are well suited toperforming various other procedures or variations of the proceduresrecited in flow diagrams 1300, 1400, and 1500. Likewise, in someembodiments, the procedures in flow diagrams 1300, 1400, and 1500 may beperformed in an order different than presented and/or not all of theprocedures described in one or more of these flow diagrams may beperformed. It is further appreciated that procedures described in flowdiagrams 1300, 1400, and 1500 may be implemented in hardware, or acombination of hardware with firmware and/or software. In someembodiments, one or more of the procedures described in flow diagrams1300, 1400, and 1500 may be implemented in response to payment of a feeand/or implemented within or via a cloud-based computing environment inresponse to payment of a fee and/or as part of a fee based service.

FIG. 13 is a flow diagram 1300 of a method of software item testing, inaccordance with various embodiments.

At procedure 1310 of flow diagram 1300, in one embodiment, an automatedtest of a software item 213 is conducted. This can be any of the testspreviously discussed and described herein, such as a test case 315, forexample. The automated test of the software item may be conductedutilizing a computing environment such as any of computing environments210A, 2108, 220, 910A, 9108 or the like, and may utilize a real orvirtual machine as a device under test.

At procedure 1320 of flow diagram 1300, in one embodiment, test dataassociated with an error condition is captured in response to anoccurrence of the error condition during a test. As previouslydescribed, the test data associated with the error condition maycomprise a capture of one or more of: a test log of actions performedduring the test (which may be annotated with a description of the errorcondition), a summary of statistics regarding the test (which may beincluded as a part of the test log), a screen shot of an execution stateof the software item upon occurrence of the error condition (e.g., ascreen shot of a GUI, such as screen shot 490), a device under test logfile (or snap shot thereof) reflective of the time of occurrence of theerror condition, a software item log file from the software item undertest (or snap shot thereof) reflective of the time of occurrence of theerror condition. In the case of a test log of actions performed duringthe test, the test log(e.g., log 477) may or may not be annotated with adescription of the error condition (e.g., an annotation in the manner ofannotation 483). Additionally, or alternatively, the test log(e.g., log477) may include a hyperlink (e.g., hyperlink 481) that provides a linkfrom a point in the test log associated with the error condition toadditional captured data regarding the error condition. Somenon-limiting examples of additional information which may be assessedvia such a hyperlink include, but are not limited to: a screen shot of aGUI at the time of the error condition, a software item log file (orsnap shot thereof) at the time of occurrence of the error condition, ora combination of a screen shot with log files.

At procedure 1330 of flow diagram 1300, in one embodiment, a portion ofthe captured test data is forwarded to one or more designated electronicnotification locations. Such electronic notification location(s) may beinput, and thus designated, by a user via a user interface such as userinterface 1000, or other user interface 212 provided by a computingenvironment (210A, 201B, 910A, 910B). Some non-limiting examples ofelectronic notification locations that all or some portion of thecaptured test data may be forwarded to include, but are not limited to,one or more of: an electronic mail address/addresses, a social network,a wiki, a flat file, a spread sheet, and a network storage location.

FIG. 14 is a flow diagram 1400 of a method of software item testing, inaccordance with various embodiments.

At procedure 1410 of flow diagram 1400, in one embodiment, an automatedtest of a software item 213 is conducted within a computing environment.This can be any of the tests previously discussed and described herein,such as a test case 315, for example. The automated test of the softwareitem may be conducted utilizing or within a computing environment suchas any of computing environments 210A, 210B, 220, 910A, 910B or thelike, and may utilize a real or virtual machine as a device under test.

At procedure 1420 of flow diagram 1400, in one embodiment, responsive tooccurrence of an error condition during the test, documentation of anexecution state associated with the software item is automaticallycaptured at a time of occurrence of the error condition. Thisdocumentation is automatically captured in response to an occurrence ofthe error condition during the test. This may comprise capture of one ormore of: a screen shot of an execution state of the software item uponoccurrence of the error condition (e.g., a screen shot of a GUI, such asscreen shot 490), a device under test log file (or snap shot thereof)reflective of the time of occurrence of the error condition, a softwareitem log file from the software item under test (or snap shot thereof)reflective of the time of occurrence of the error condition.

At procedure 1430 of flow diagram 1400, in one embodiment, the captureddocumentation of the execution state associated with the software itemis automatically electronically forwarded to one or more designatedelectronic notification locations. Such electronic notificationlocation(s) may be input, and thus designated, by a user via a userinterface such as user interface 1000, or other user interface 212provided by a computing environment (210A, 201B, 910A, 910B). Somenon-limiting examples of electronic notification locations that all orsome portion of the captured documentation of the execution state may beforwarded to include, but are not limited to, one or more of: anelectronic mail address/addresses, a social network, a wiki, a flatfile, a spread sheet, and a network storage location. The actualcaptured documentation may be forwarded, a hyperlink to the captureddocumentation of the execution state may be forwarded, or a hyperlink tothe captured documentation of the execution state may be included in adocument that is forwarded. In one embodiment, for example, a hyperlink(or multiple hyperlinks) to captured documentation may be embedded in adocument, such as a log 477 of testing actions performed during thetest, and the document may be electronically forwarded to one or moredesignated electronic notification locations. Hyperlink 481 in FIG. 4Gis one example of such an embedded hyperlink.

FIG. 15 is a flow diagram 1500 of a method of software item testing, inaccordance with various embodiments.

At procedure 1510 of flow diagram 1500, in one embodiment, an automatedtest of a software item 213 is conducted. This can be any of the testspreviously discussed and described herein, such as a test case 315, forexample. The automated test of the software item may be conductedutilizing or within a computing environment such as any of computingenvironments 210A, 210B, 220, 910A, 910B or the like, and may utilize areal or virtual machine as a device under test.

At procedure 1520 of flow diagram 1500, in one embodiment, an errorcondition occurs during the testing and, in response to the occurrenceof the error condition, a plurality of procedures 1522, 1524, and 1526are performed.

At procedure 1522 of flow diagram 1500, in one embodiment, a descriptionof the error condition is automatically annotated into a test log ofactions performed during the test. This annotation creates an annotatedtest log. With reference to FIG. 4G, log 477 includes annotation 483 andis an example of an annotated test log.

At procedure 1524 of flow diagram 1500, in one embodiment, documentationof an execution state associated with the software item is automaticallycaptured at a time of occurrence of the error condition. Thisdocumentation is automatically captured in response to an occurrence ofthe error condition during the test and may comprise a capture of one ormore of: a screen shot of an execution state of the software item uponoccurrence of the error condition (e.g., a screen shot of a GUI, such asscreen shot 490), a device under test log file (or snap shot thereof)reflective of the time of occurrence of the error condition, a softwareitem log file from the software item under test (or snap shot thereof)reflective of the time of occurrence of the error condition.

At procedure 1526 of flow diagram 1500, in one embodiment, a hyperlinkto the captured documentation is automatically embedded within theannotated test log.

In one embodiment, for example, a hyperlink (or multiple hyperlinks) tocaptured documentation may be embedded in a document, such as a log 477of testing actions performed during the test, and the document may beelectronically forwarded to one or more designated electronicnotification locations. Hyperlink 481 in FIG. 4G is one example of suchan embedded hyperlink which electronically references captureddocumentation. Such a hyperlink allows an individual to easily andquickly access additional information about an error condition whilereviewing a test log.

In one embodiment, the flow diagram as described in procedures 1510 and1520 further includes automatically electronically forwarding theannotated test log to one or more designated electronic notificationlocations. Such electronic notification location(s) may be input, andthus designated, by a user via a user interface such as user interface1000, or other user interface 212 provided by a computing environment(210A, 201B, 910A, 910B). Some non-limiting examples of electronicnotification locations that the annotated test log may be forwarded toinclude, but are not limited to, one or more of: an electronic mailaddress/addresses, a social network, a wiki, a flat file, a spreadsheet, and a network storage location.

Software Development Kit Testing

Typically, specialized knowledge and expertise is required to properlytest a software development kit (SDK). Testing an SDK typically involvesdeveloping a basic application to invoke the exposed methods of the SDK,and expose callback methods from the SDK. Conventionally, each time adeveloper needs to test a different aspect of the SDK, a code change andre-compile of the test application is required. Embodiments describedherein eliminate the requirement for such re-compiling and allow forcreating tests using natural language rather than a programminglanguage.

FIG. 16 is a block diagram of an example testing framework 1008,according to various embodiments. Testing framework 1008 is similar totesting framework 100 except that product fixture layer 130 has beenreplaced with SDK fixture layer 1630. In one embodiment, using testingframework 1008, SDK fixture layer 1630 comprises an action for eachentry point into an SDK under test. In an embodiment, each entry pointis invoked by providing the specific parameters an entry point requires.As a user writes a test using testing framework 1008, they are actuallywriting an application. For example, an application may be created whenan SDK test is run. In various examples, that application mayconsume/exercise/utilize the SDK. In various embodiments, an applicationis thus consuming/exercising/utilizing the SDK in a manner similar tothat of a typical Original Equipment Manufacturer's (OEM) applicationthat uses the SDK. In some embodiments, each variation of a testeffectively comprises a different application (equitable to an OEMapplication that uses the SDK), able to invoke any entry point in anysequence. In various embodiments, an output from the SDK is verified.

In various embodiments, both positive and negative tests may bevalidated. A negative test is a test where a failure is the expectedoutput. For example, failures may be verified, and the handling of thosefailures may be verified. A positive test is a test where a success isthe expected output. In some embodiments, a test comprises a combinationof both positive and negative tests. In other embodiments, only apositive test or only a negative test is performed.

In some embodiments, a user may describe actions using an SDKapplication programming interface (API). In other words, usingembodiments described herein, a test author can write test cases usingan API manual. Thus, the natural language is bound to the API.

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIG. 17,flow diagram 1700 illustrates example procedures used by variousembodiments. Flow diagram 1700 includes some procedures that, in variousembodiments, are carried out by a processor under the control ofcomputer-readable and computer-executable instructions. In this fashion,procedures described herein and in conjunction with flow diagram 1700are, or may be, implemented using a computer, in various embodiments.The computer-readable and computer-executable instructions can reside inany tangible computer readable storage media. Some non-limiting examplesof tangible computer readable storage media include random accessmemory, read only memory, magnetic disks, and optical disks, any or allof which may be employed with a computing environment (210A, 210B, 910A,910B, 220). The computer-readable and computer-executable instructions,which reside on tangible computer readable storage media, are used tocontrol or operate in conjunction with, for example, one or somecombination of processors of the computing environment (201A, 210B,910A, 910B, 220). It is appreciated that the processor(s) may bephysical or virtual or some combination thereof (it should also beappreciated that a virtual processor is implemented on physicalhardware). Although specific procedures are disclosed in flow diagram1700, such procedures are examples. That is, embodiments are well suitedto performing various other procedures or variations of the proceduresrecited in flow diagram 1700. Likewise, in some embodiments, theprocedures in flow diagram 1700 may be performed in an order differentthan presented and/or not all of the procedures described in one or moreof these flow diagrams may be performed. It is further appreciated thatprocedures described in flow diagram 1700 may be implemented inhardware, or a combination of hardware with firmware and/or software. Insome embodiments, one or more of the procedures described in flowdiagram 1700 may be implemented in response to payment of a fee and/orimplemented within or via a cloud-based computing environment inresponse to payment of a fee and/or as part of a fee-based service.

FIG. 17 is a flow diagram 1700 of a method of software development kittesting, in accordance with various embodiments.

At procedure 1710 of flow diagram 1700, in one embodiment, a graphicaluser interface is presented for initializing a test of an SDK. In anembodiment, graphical user interface generator 211, comprised within atask scheduling computing environment 910A or a task schedulingcloud-based computing environment 910B, provides a user interface 212comprising test information and/or actions.

At procedure 1720 of flow diagram 1700, in one embodiment, anapplication is created to run against the SDK.

At procedure 1725 of flow diagram 1700, in one embodiment, naturallanguage is bound to an application programming interface. For example,rather than learning a programming language to write code for testingthe SDK, the same commonly used for writing test cases 315 may be usedto write the tests for the SDK.

At procedure 1730 of flow diagram 1700, in one embodiment, an output ofthe SDK is verified. The output may be a positive or a negative output.For instance, a failure may be expected, and as such the test willverify a failure occurred. Moreover, a test may verify that the failurewas handled correctly by an error handler.

At procedure 1740 of flow diagram 1700, in one embodiment, whenincluded, each entry point into an SDK is invoked. For example, anapplication may be written to invoke each entry point into an SDK.

At procedure 1760 of flow diagram 1700, in one embodiment, whenincluded, interactions are tested with the SDK. As discussed herein,positive and negative interactions may be tested with an SDK. In someembodiments, the output of an SDK is expected to be positive. In otherembodiments, the output of an SDK is expected to be negative. Moreover,in some embodiments an error handler is expected to operate in aparticular manner. In various embodiments the SDK test is operable toverify that an error handler performed correctly.

At procedure 1770 of flow diagram 1700, in one embodiment, whenincluded, callbacks from the SDK back to entry points in an applicationare tested. In one embodiment, parameter values generated by the SDKpassed to the callbacks are tested. In one embodiment, when included, asequence of callbacks that an SDK makes to the application are tested.For example, a test may verify that event A occurs before event B, andthat event B occurs before event C. In one embodiment, testing that theSDK does not make particular callbacks to the application is performed.For example, a test may determine whether or not a particular eventoccurred. In one embodiment, testing a set of callbacks from the SDK tothe application without regard to a sequence is performed. For example,event A may have occurred five times, event B may have occurred fourtimes, event C may have occurred zero times, and any number of any otherevents may have occurred.

Test Environment Managed Within Tests

In some embodiments described herein, test frameworks (100, 100B) managetest cases 315, test execution, and test infrastructure. In otherembodiments discussed herein, a test framework 100C (FIG. 18) isutilized in which tests themselves are responsible for managing the testenvironment they will be executing the test against. For example, whentest collateral exists in a plurality of target computing environmentvirtual machines (e.g., two, dozens, hundreds, or thousands of virtualmachines), managing the test environment within the plurality of virtualmachines allows one or more test cases 315 to run within the same testenvironment across the dozen of machines. In various embodiments, thisincludes reverting virtual machine snapshots, and/or starting andstopping virtual machines. In some embodiments, a test does not knowbeforehand, the specific details of the testing environment it will berunning in. Thus, in some examples providing a test environment (e.g.,test environment 1920 of FIGS. 19A and 19B) a test is allowed to run ina predetermined testing environment.

When managing a testing environment, certain modifications may beimplemented which are not in test cases 315, as described elsewhereherein. For instance, modification may exist with respect to test aparticular environment that may need to be stopped and then restarted.In some embodiments, snap shots are taken of a virtual environment andsent back to a test. In various embodiments, commands and/or actions foraltering a testing environment are written in the same commonly usedlanguage as custom actions 320 and/or as test cases 315

FIG. 18 shows an example test framework 100C comprising a testenvironment management add-on 1830. Test framework 100C is similar totesting framework 100 except that environment management add-on 1830provides additional actions available to all layers. Similar tofixtures, add-ons are simply a collection of actions of a particulartype, which are made available for use by all fixtures, including basefixtures and product fixtures. In an embodiment, test environmentmanagement add-on 1830 is operable to modify the testing environment1920 it is responsible for testing.

For tests to run against any test environment 1920, in one embodiment,the use of at least one global variable file 1930 that describes thetest environment 1920 is utilized. Tests may use generic references tomachine names, which are defined in a global variable file 1930 (e.g.,an environment file). If a portion of underlying infrastructure ischanged (e.g., a global variable file 1930), the change impacts everytest that utilizes that portion of underlying infrastructure.

In one embodiment, a test is vendor agnostic. In other words, a testframework 100C that is capable of modifying test environments 1920 mayoperate on any type of machine, operating system, API, application, etc.In other embodiments, a test framework 100C is designed to operate on aparticular type of virtual machine, operating system, application, etc.

In one embodiment, a graphical user interface (GUI) is presented toinitialize a test. In one embodiment, the same (or another) GUI ispresented for the entry of parameters for a test environment 1920. Forexample, a GUI may be presented such that a user can enter a command toshut off, revert, or start a particular virtual machine.

FIGS. 19A and 19B are block diagrams of example computing environments(1910A, 1910B) with which or upon which embodiments may be implemented.

FIG. 19A is similar to FIG. 9A, with the additions of testingenvironment 1920 and global variable file 1930, while FIG. 19B issimilar to FIG. 9B, with the additions of testing environment 1920 andglobal variable file 1930.

In some embodiments, some or all of the actions and processes describedherein, with respect to a test environment being managed within a test,are performed on a physical computing environment 1910A. In someembodiments, some or all of the actions and processes described herein,with respect to a test environment being managed within a test, areperformed in a cloud-based environment such as cloud-based computingenvironment 1910B.

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIG. 20,flow diagram 2000 illustrates example procedures used by variousembodiments. Flow diagram 2000 includes some procedures that, in variousembodiments, are carried out by a processor under the control ofcomputer-readable and computer-executable instructions. In this fashion,procedures described herein and in conjunction with flow diagram 2000are, or may be, implemented using a computer, in various embodiments.The computer-readable and computer-executable instructions can reside inany tangible computer readable storage media. Some non-limiting examplesof tangible computer readable storage media include random accessmemory, read only memory, magnetic disks, and optical disks, any or allof which may be employed with a computing environment (210A, 201B, 910A,910B, 1910A, 1910B, 220). The computer-readable and computer-executableinstructions, which reside on tangible computer readable storage media,are used to control or operate in conjunction with, for example, one orsome combination of processors of the computing environment (210A, 201B,910A, 910B, 1910A, 1910B, 220). It is appreciated that the processor(s)may be physical or virtual or some combination thereof (it should alsobe appreciated that a virtual processor is implemented on physicalhardware). Although specific procedures are disclosed in flow diagram2000, such procedures are examples. That is, embodiments are well suitedto performing various other procedures or variations of the proceduresrecited in flow diagram 2000. Likewise, in some embodiments, theprocedures in flow diagram 2000 may be performed in an order differentthan presented and/or not all of the procedures described in one or moreof these flow diagrams may be performed. It is further appreciated thatprocedures described in flow diagram 2000 may be implemented inhardware, or a combination of hardware with firmware and/or software. Insome embodiments, one or more of the procedures described in flowdiagram 2000 may be implemented in response to payment of a fee and/orimplemented within or via a cloud-based computing environment inresponse to payment of a fee and/or as part of a fee based service.

FIG. 20 is a flow diagram 2000 of a method of managing a testenvironment, in accordance with various embodiments.

At procedure 2010 of flow diagram 2000, in one embodiment, a test isinitialized. This can comprise presentation of a GUI via which the testis may be initialized. In an embodiment, the initialized test managesthe test environment 1920 that the test executes test cases 315 againstduring run time. In one embodiment, a test is performed on a physicalcomputing environment 1910A; while in another embodiment a test isperformed in a cloud-based computing environment 1910B. In some cases, atest may be performed on both a physical computing environment 1910A anda cloud-based computing environment 1910B.

In one embodiment, the test modifies a test environment 1920 such thatthe test begins in a predetermined test environment 1920. For example,if a test requires a virtual machine to be shut off, the test willmodify test environment 1920 such that the virtual machine is shut off.If a test requires a virtual machine to be powered on at a specificsnapshot, the test will modify test environment 1920 such that thevirtual machine is reverted to the specific snapshot and powered on. Ifa test requires a plurality of virtual machines in various operationalstates, the test will modify test environment 1920 such that theplurality of virtual machines be in their required initialized state.

In one embodiment the test which manages test environment 1920 iswritten in the same commonly used language as a test that does notmanage a test environment 1920

At procedure 2020 of flow diagram 2000, in one embodiment, at least oneglobal variable file is accessed by the test. In an embodiment, a globalvariable file 1930 describes test environment 1920. It should beappreciated that in a cloud-based computing environment 1910B, a testcan dynamically acquire, create and/or provision the required systemsand likewise dynamically dispose of the systems upon test completion. Itshould also be made clear that there is no direct dependency or bindingto any specific instance of a system required to perform a test. Aglobal variable adds a layer of abstraction, wherein a user may use ageneric reference to a machine, or plurality of machines, and globalvariable file 1930 will assist in translating the machine referencesinto specific machines available in the test environment 1920.

At procedure 2030 of flow diagram 2000, in one embodiment, whenincluded, virtual machine snapshots are reverted. In an embodiment,snapshots of a virtual machine are reverted such that the test mayguarantee the virtual machines are in a known state. In an embodiment,the virtual machine may be a target computing environment such as targetcomputing environment 220-1 of FIGS. 19A and 19B.

At procedure 2040 of flow diagram 2000, in one embodiment, whenincluded, at least one virtual machine is started. For example, managinga testing environment 1920 may include tests that require at least onevirtual machine (e.g., 1910A, 1910B, 220) to be starting up and/orrecently started in order to perform a test.

At procedure 2050 of flow diagram 2000, in one embodiment, whenincluded, at least one virtual machine is stopped. For example, managinga testing environment 1920 may include tests that require at least onevirtual machine (e.g., 1910A, 1910B, 220) to be stopped in order toperform a test.

Test Creation with Execution

In various embodiments, framework 100 comprises a test editor (e.g.,custom testing action GUI 410, custom action editor 440, etc.) thatperforms real time validation and execution of a test while the test isbeing written.

With references again to FIGS. 4A-4F, in some embodiments, as a userenters custom actions 320 into a line 425 in a test editor, the lines425 are continuously being verified for syntax errors 450 andcompleteness. In other words, after a portion of a parameter 426 isentered into a test (e.g., test case 315), that portion of a parameter426 is tested for syntax errors 450 and completeness. If a syntax error450 is found, an indication is provided such as an exclamation point asshown in FIG. 4E. In some embodiments, syntax verification and testexecution are performed upon the entry of an action 421.

In one embodiment, once a line 425 is entered (i.e., complete) and freefrom syntax errors 450, a user may execute that line 425 (e.g., executethat custom action 320) via the test editor to verify the intendedlogic. In some embodiments, execution is performed automatically after aline 425 is found to be free from syntax errors 450. A user may executemultiple lines 425 to verify the intended logic for each line 425 in atest. In one example, a test will indicate whether it will executesuccessfully and is free from syntax errors 450 and logic errors, thusreducing test creation time. It should be appreciated that, via the testeditor a user may select any line 425 or combination of lines 425 forexecution at any time, without restriction.

In one embodiment, when every line 425 of a test is executed in order toverify logic, the entire test is run. In other words, in addition toeach line 425 being verified by execution, each action 421 is alsoperformed when executed. For example, FIG. 4C shows a custom actioneditor 440. In the first row, the action 421 “WaitWindowExit” has beenverified and executed. When verifying whether action 421 “Click”performs the correct action, it is also executed (i.e., run). As anotherexample, FIG. 4C shows a plurality of lines 425. If the lines 425comprise an entire test, by verifying each line 425 in order, the entiretest has been run. In one embodiment, an indication of a completed testis presented if a test is complete after every line 425 is entered andverified (i.e., executed).

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIG. 21,flow diagram 2100 illustrates example procedures used by variousembodiments. Flow diagram 2100 includes some procedures that, in variousembodiments, are carried out by a processor under the control ofcomputer-readable and computer-executable instructions. In this fashion,procedures described herein and in conjunction with flow diagram 2100are, or may be, implemented using a computer, in various embodiments.The computer-readable and computer-executable instructions can reside inany tangible computer readable storage media. Some non-limiting examplesof tangible computer readable storage media include random accessmemory, read only memory, magnetic disks, and optical disks, any or allof which may be employed with a computing environment (910A, 910B,1910A, 1910B, 220). The computer-readable and computer-executableinstructions, which reside on tangible computer readable storage media,are used to control or operate in conjunction with, for example, one orsome combination of processors of the computing environment (910A, 910B,1910A, 1910B, 220). It is appreciated that the processor(s) may bephysical or virtual or some combination thereof (it should also beappreciated that a virtual processor is implemented on physicalhardware). Although specific procedures are disclosed in flow diagram2100, such procedures are examples. That is, embodiments are well suitedto performing various other procedures or variations of the proceduresrecited in flow diagram 2100. Likewise, in some embodiments, theprocedures in flow diagram 2100 may be performed in an order differentthan presented and/or not all of the procedures described in one or moreof these flow diagrams may be performed. It is further appreciated thatprocedures described in flow diagram 2100 may be implemented inhardware, or a combination of hardware with firmware and/or software. Insome embodiments, one or more of the procedures described in flowdiagram 2100 may be implemented in response to payment of a fee and/orimplemented within or via a cloud-based computing environment inresponse to payment of a fee and/or as part of a fee based service.

FIG. 21 is a flow diagram 2100 of a method of executing a test whilecreating the test, in accordance with various embodiments.

At procedure 2110 of flow diagram 2100, in one embodiment, a portion ofa line 425 of a test case 315 for a software item is received.

At procedure 2120 of flow diagram 2100, in one embodiment, syntax isvalidated upon entry of a portion of a line 425. In one embodiment, theparameters 426 entered into line 425 are tested for syntax errors 450upon the entry of each character. In one embodiment, an indication of anerror based on syntax 450 is presented prior to execution of said line425 in said test case 315.

At procedure 2130 of flow diagram 2100, in one embodiment, a line 425 isexecuted after the syntax of that line is validated. For example, uponentry of a custom action 320, the custom action is executed/run. In anembodiment, a line 425/custom action 320 is executed automatically afterthe line 425/custom action 320 passes a test for syntax errors 450. Inanother embodiment, a line 425 is executed after a user chooses toexecute the line 425.

At procedure 2140 of flow diagram 2100, in one embodiment, any errorbased on execution is indicated. This can include, but is not limitedto, indicating an error based on syntax. For example, in someembodiments, an exclamation point or an error symbol is presented upondetermining the presence of a syntax error 450.

At procedure 2150 of flow diagram 2100, in one embodiment, whenincluded, any error that occurs while executing a line 425 is indicated.For example, if a line 425 references a GUI that is not available uponexecution of that line, an indication that the GUI is not available(e.g., 483 of FIGS. 4G and 4H) is presented.

Test Language Interpreter

Typically, conventional user interface automation is difficult to writeand costly to maintain. However, embodiments of framework 100 discussedherein provide for user interface changes, including, but not limitedto: moving elements, renaming elements, changing text, etc. In anembodiment, a language evaluator interprets an entry, and then performsuser interface tree 2210 (FIG. 22) searches by combining a plurality ofuser interface properties for an intuitive user automation experience.Moreover, in an embodiment, debugging a user interface 212 does not needto be performed before a search.

A language evaluator 2200 may identify a control (e.g., an edit control)by using the name of the label control adjacent to the edit control, andperform multi-step user interface tree 2010 searches for controls buriedinside of panes, by referencing the appearance of the control instead ofthe unique identification of the control.

In an embodiment, a user may enter plain English search term 2230 intolanguage evaluator 2200. Using plain English term, 2230, a linkage isbuilt at run time to a control 2220. This way, a user does not need toknow the underlying structure of an application. A user may access acontrol 2220 of an application simply by describing a control 2220.

In an embodiment, language evaluator 2200 combines parameters 2240 topin a control in the software item under test 213. Parameters include,but are not limited to: type, label, appearance, size, associated text,location on screen, etc. In one example, a tree 2210 of controls issearched for controls 2220 using a combination of parameters. In otherwords, language evaluator 2200 can find a control 2220 based on adescription of the appearance of that control 2220.

In one embodiment, a tree 2210 is searched for a control 2220 in a userinterface 212 in real time. For example, a test may click on “menu,”then “file,” then “new,” then finally “machine group.” Typically, codingthis action would require knowledge of performing tree 2210 searches. Inan embodiment, by entering plain English words, the tree 2210 issearched. In this example, language evaluator will search tree 2210 andfind “machine group” within “new,” which is within “file,” which iswithin “menu.”

The following discussion sets forth in detail the operation of someexample methods of operation of embodiments. With reference to FIG. 23,flow diagram 2300 illustrates example procedures used by variousembodiments. Flow diagram 2300 includes some procedures that, in variousembodiments, are carried out by a processor under the control ofcomputer-readable and computer-executable instructions. In this fashion,procedures described herein and in conjunction with flow diagram 2300are, or may be, implemented using a computer, in various embodiments.The computer-readable and computer-executable instructions can reside inany tangible computer readable storage media. Some non-limiting examplesof tangible computer readable storage media include random accessmemory, read only memory, magnetic disks, and optical disks, any or allof which may be employed with a computing environment (210A, 210B, 910A,910B, 1910A, 1910B, 220). The computer-readable and computer-executableinstructions, which reside on tangible computer readable storage media,are used to control or operate in conjunction with, for example, one orsome combination of processors of the computing environment (210A, 210B,910A, 910B, 1910A, 1910B, 220). It is appreciated that the processor(s)may be physical or virtual or some combination thereof (it should alsobe appreciated that a virtual processor is implemented on physicalhardware). Although specific procedures are disclosed in flow diagram2300, such procedures are examples. That is, embodiments are well suitedto performing various other procedures or variations of the proceduresrecited in flow diagram 2300. Likewise, in some embodiments, theprocedures in flow diagram 2300 may be performed in an order differentthan presented and/or not all of the procedures described in one or moreof these flow diagrams may be performed. It is further appreciated thatprocedures described in flow diagram 2300 may be implemented inhardware, or a combination of hardware with firmware and/or software. Insome embodiments, one or more of the procedures described in flowdiagram 2000 may be implemented in response to payment of a fee and/orimplemented within or via a cloud-based computing environment inresponse to payment of a fee and/or as part of a fee based service.

FIG. 23 is a flow diagram 2300 of a method of binding a software item toa plain English name, in accordance with various embodiments.

At procedure 2310 of flow diagram 2300, in one embodiment, a user inputof a plain English control name is received into a test file. In someembodiments, a user input is entered into a search bar. In otherembodiments, a user input may comprise a selection from a list.

At procedure 2320 of flow diagram 2300, in one embodiment, a userinterface under test is automatically searched for a controlcorresponding to the plain English control name. In one embodiment,searching a user interface 212 comprises traversing a tree 2210 ofcontrols. In one embodiment, searching a user interface 212 comprisesscanning parameters 2240 within trees 2210. For example, while scanninga tree 2210, parameters within an item in the tree 2210 may be scannedto determine whether the control 2220 has been found. In one embodiment,debugging does not need to be performed before a search. In someexamples, a user interface 212 must be debugged before a control 2220can be bound to a plain English name.

At procedure 2330 of flow diagram 2300, in one embodiment, a control inthe software item under test is dynamically bound to the plain Englishcontrol name. In one embodiment, binding a plain English control name toa software item under test comprises determining the location of acontrol 2220. In one example, searching and/or binding occur in realtime.

CONCLUSION

Numerous embodiments have been described herein as examples, and manyembodiments are also more fully described under distinct headings.Embodiments were presented in this manner for purposes of clarity, andsuch presentation is not meant to imply that embodiments described inisolation from other embodiments are not interoperable. To the contrary,it should be appreciated that the embodiments described herein may becombined in a variety of combinations with one another and that suchcombinations of the embodiments described herein are fully anticipated.Some non-limiting examples of such combinations of the embodiments andconcepts describe herein include, but are not limited to: combiningembodiments discussed in the Custom Actions section with embodimentsdiscussed in the Scheduled Software Item Testing section; combiningembodiments discussed in the Custom Editors section with embodimentsdiscussed in the Altering a Test while a Test is in an Active Statesection, further combining with embodiments discussed in the SoftwareDevelopment Kit Testing section; etc.

Example embodiments of the subject matter are thus described. Althoughvarious embodiments have been described in a language specific tostructural features and/or methodological acts, it is to be understoodthat the appended claims are not necessarily limited to the specificfeatures or acts described above. Rather, the specific features and actsdescribed above are disclosed as example forms of implementing theclaims and their equivalents.

What is claimed is:
 1. A non-transitory computer readable storage mediumcomprising instructions stored thereon which, when executed, cause acomputing system to perform a method of software item testing, saidmethod comprising: conducting an automated test of a software item;responsive to occurrence of an error condition during said test,capturing test data associated with said error condition; and forwardinga portion of said captured test data to a designated electronicnotification location.
 2. The non-transitory computer readable storagemedium of claim 1, wherein said conducting an automated test of asoftware item comprises: conducting said automated test of said softwareitem utilizing a virtual machine as a device under test.
 3. Thenon-transitory computer readable storage medium of claim 1, wherein saidconducting an automated test of a software item comprises: conductingsaid automated test of said software item utilizing a cloud computingenvironment.
 4. The non-transitory computer readable storage medium ofclaim 1, wherein said capturing test data associated with said errorcondition comprises: capturing a screen shot of an execution state ofsaid software item upon occurrence of said error condition.
 5. Thenon-transitory computer readable storage medium of claim 1, wherein saidcapturing test data associated with said error condition comprises:capturing a snap shot of a device under test log file upon occurrence ofsaid error condition.
 6. The non-transitory computer readable storagemedium of claim 1, wherein said capturing test data associated with saiderror condition comprises: capturing a snap shot of software item logfile from said software item under test upon occurrence of said errorcondition.
 7. The non-transitory computer readable storage medium ofclaim 1, wherein said capturing test data associated with said errorcondition comprises: creating a summary of statistics regarding saidtest.
 8. The non-transitory computer readable storage medium of claim 1,wherein said capturing test data associated with said error conditioncomprises: capturing a test log of actions performed during said test.9. The non-transitory computer readable storage medium of claim 8,wherein said capturing test data associated with said error conditioncomprises: annotating said test log with a description of said errorcondition.
 10. The non-transitory computer readable storage medium ofclaim 9, wherein said capturing test data associated with said errorcondition comprises: embedding a hyperlink in said test log at a pointassociated with said error condition, said hyperlink comprising a linkto additional captured data regarding said error condition.
 11. Thenon-transitory computer readable storage medium of claim 1, wherein saidforwarding a portion of said captured test data to a designatedelectronic notification location comprises: forwarding said portion ofsaid captured test data to a designated electronic notification locationselected from the group consisting of: an electronic mail address, asocial network, a wiki, a flat file, a spread sheet, and a networkstorage location.
 12. A method of software item testing, said methodcomprising: conducting an automated test of a software item within acomputing environment; responsive to occurrence of an error conditionduring said test, automatically capturing documentation of an executionstate associated with said software item at a time of occurrence of saiderror condition; and electronically forwarding said captureddocumentation of the execution state associated with said software itemto a designated electronic notification location.
 13. The method asrecited in claim 12, wherein said automatically capturing documentationof an execution state associated with said software item at a time ofoccurrence of said error condition comprises: capturing at least one ofa screen shot of graphical user interface of said software item, a snapshot of a software item testing log from said software item, and a snapshot of a device under test testing log from a computing environmentupon which said software item is being tested.
 14. The method as recitedin claim 12, wherein said electronically forwarding said captureddocumentation of an execution state associated with said software itemto a designated electronic notification location comprises:electronically forwarding said captured documentation to a designatedelectronic notification location selected from the group consisting of:an electronic mail address, a social network, a wiki, a flat file, aspread sheet, and a network storage location.
 15. The method as recitedin claim 12, wherein said electronically forwarding said captureddocumentation of an execution state associated with said software itemto a designated electronic notification location comprises:electronically referencing said captured documentation as a hyperlinkwithin a test log of actions performed during said test.
 16. A method ofsoftware item testing, said method comprising: conducting an automatedtest of a software item within a computing environment; and responsiveto occurrence of an error condition during said test: automaticallyannotating a description of said error condition into a test log ofactions performed during said test to create an annotated test log;automatically capturing documentation of an execution state associatedwith said software item at a time of occurrence of said error condition;and embedding a hyperlink to said captured documentation within saidannotated test log.
 17. The method as recited in claim 16, furthercomprising: electronically forwarding said annotated test log to adesignated electronic notification location.
 18. The method as recitedin claim 17, wherein said electronically forwarding said log of actionsto a designated electronic notification location comprises:electronically forwarding said annotated test log to a designatedelectronic notification location selected from the group consisting of:an electronic mail address, a social network, a wiki, a flat file, aspread sheet, and a network storage location.